Microscope system, superimposing unit, and operation method

ABSTRACT

A microscope system includes: a microscope optical system that includes an ocular lens and forms an optical image of a sample on an object side of the ocular lens; a processor that generates auxiliary image data based on information regarding a target slide selected from among a plurality of ordered slides included in a slide set; and a superimposing device that superimposes, based on the auxiliary image data, an auxiliary image including the target slide on an image plane on which the optical image is formed. The processor selects, in response to an instruction to switch the target slide, a slide determined according to a first order in which the plurality of slides are ordered as a new target slide.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromprior Japanese Patent Application No. 2022-035427, filed Mar. 8, 2022,the entire contents of which are incorporated herein by this reference.

TECHNICAL FIELD

The present disclosure relates to a microscope system, a superimposingunit, and an operation method.

BACKGROUND

Even at present when automation of work by a robot or the likeprogresses, there are many products required to be manually assembled. Amedical device is an example of the products. Assembly of precisiondevices such as medical devices is often performed under a microscopebecause of a lot of detailed work, and a stereomicroscope capable ofstereoscopically viewing an object with both eyes is often used.

However, in order to check a procedure manual during the assembly workwhile observing the object with the stereomicroscope, the eyes need tobe temporarily separated from an ocular lens of the stereomicroscope,and the line of sight needs to be moved to a display or the like onwhich the procedure manual is displayed. Then, after the checking, theassembly work is continued by looking into the ocular lens again, sothat the work efficiency is hardly increased.

A technique related to such a problem is described in, for example, WO2020/066041 A. In a system described in WO 2020/066041 A, by projectingan image at an intermediate image position of a microscope, necessaryinformation can be obtained while looking into an ocular lens.

SUMMARY

A microscope system according to an aspect of the present inventionincludes: a microscope optical system that includes an ocular lens andforms an optical image of a sample on an object side of the ocular lens;a processor that generates auxiliary image data based on informationregarding a target slide selected from among a plurality of orderedslides included in a slide set; and a superimposing device thatsuperimposes, based on the auxiliary image data, an auxiliary imageincluding the target slide on an image plane on which the optical imageis formed. The processor selects, in response to an instruction toswitch the target slide, a slide determined according to a first orderin which the plurality of slides are ordered as a new target slide.

A superimposing unit according to another aspect of the presentinvention is attached to a microscope including a microscope opticalsystem that forms an optical image of a sample on an object side of anocular lens, the superimposing unit including: a processor thatgenerates auxiliary image data based on information regarding a targetslide selected from among a plurality of ordered slides included in aslide set; and a superimposing device that superimposes, based on theauxiliary image data, an auxiliary image including the target slide onan image plane on which the optical image is formed. The processorselects, in response to an instruction to switch the target slide, aslide determined according to a first order in which the plurality ofslides are ordered as a new target slide.

An operation method according to an aspect of the present invention isan operation method of a control device that controls a microscopeincluding a superimposing device and a microscope optical system thatforms an optical image of a sample on an object side of an ocular lens,the operation method including: causing a processor of the controldevice to select, in response to an instruction to switch a target slideselected from among a plurality of ordered slides included in a slideset, a slide determined according to a first order in which theplurality of slides are ordered as a new target slide; causing theprocessor to generate auxiliary image data based on informationregarding the new target slide; and causing the superimposing device tosuperimpose, based on the auxiliary image data, an auxiliary imageincluding the new target slide on an image plane on which the opticalimage is formed.

BRIEF DESCRIPTION OF DRAWINGS

The present invention will be more apparent from the following detaileddescription when the accompanying drawings are referenced.

FIG. 1 is a diagram illustrating a microscope system according to anembodiment of the present invention;

FIG. 2 is a diagram illustrating a configuration of an optical systemincluded in the microscope system;

FIG. 3 is a diagram illustrating a configuration of a slide set;

FIG. 4 is an example of a flowchart of an image projection process to beperformed by the microscope system;

FIG. 5 is a diagram for describing a configuration of an image formed onan image plane;

FIG. 6 is a diagram illustrating an example of a home screen displayedon a monitor;

FIG. 7 is a diagram illustrating an example of a screen for setting amicroscope configuration;

FIG. 8 is a diagram illustrating an example of a screen for adjusting ARdisplay;

FIG. 9 is a diagram for describing a method of adjusting the AR display;

FIG. 10 is a diagram illustrating an example of a screen for adjusting azoom sensor;

FIG. 11 is a diagram illustrating an example of a home screen observedfrom an ocular lens;

FIG. 12 is an example of a flowchart of an assembly work supportprocess;

FIG. 13 is a diagram illustrating an example of a superimposed imageobserved from the ocular lens;

FIG. 14 is a diagram for describing a slide switching operation;

FIG. 15 is a diagram illustrating another example of the superimposedimage observed from the ocular lens;

FIG. 16 is a diagram illustrating still another example of thesuperimposed image observed from the ocular lens;

FIG. 17 is a diagram illustrating still another example of thesuperimposed image observed from the ocular lens;

FIG. 18 is a diagram illustrating still another example of thesuperimposed image observed from the ocular lens;

FIG. 19 is a diagram illustrating still another example of thesuperimposed image observed from the ocular lens;

FIG. 20 is a diagram illustrating still another example of thesuperimposed image observed from the ocular lens;

FIG. 21 is a diagram illustrating still another example of thesuperimposed image observed from the ocular lens;

FIG. 22 is a diagram illustrating still another example of thesuperimposed image observed from the ocular lens;

FIG. 23 is a diagram illustrating still another example of thesuperimposed image observed from the ocular lens;

FIG. 24 is a diagram illustrating still another example of thesuperimposed image observed from the ocular lens;

FIG. 25 is a diagram illustrating still another example of thesuperimposed image observed from the ocular lens;

FIG. 26 is a diagram illustrating still another example of thesuperimposed image observed from the ocular lens;

FIG. 27 is a diagram illustrating still another example of thesuperimposed image observed from the ocular lens;

FIG. 28 is a diagram illustrating still another example of thesuperimposed image observed from the ocular lens;

FIG. 29 is a diagram illustrating an example of an imaging screenobserved from the ocular lens;

FIG. 30 is a diagram illustrating an example of an imaging screenobserved on a monitor;

FIG. 31 is a diagram illustrating an example of a recording image;

FIG. 32 is a diagram illustrating another example of the recordingimage;

FIG. 33 is a diagram illustrating still another example of the recordingimage;

FIG. 34 is a diagram for describing an image recording method;

FIG. 35 is a diagram illustrating a configuration instructed by atrainer from a remote location;

FIG. 36 is an example of a flowchart of a process of creating a slideset;

FIG. 37 is a diagram illustrating an example of a screen for setting aparameter of a slide file;

FIG. 38 is a diagram illustrating an example of a slide setting screen;

FIG. 39 is a diagram illustrating a hardware configuration of a computerfor implementing a control device;

FIG. 40 is a diagram illustrating still another example of thesuperimposed image observed from the ocular lens; and

FIG. 41 is a diagram illustrating still another example of thesuperimposed image observed from the ocular lens.

DESCRIPTION OF EMBODIMENTS

By the way, in a case where assembly work including a plurality ofprocesses is assumed, it is desirable to appropriately switch anddisplay appropriate information at a necessary timing, instead ofdisplaying specific information at all times.

In view of the above circumstances, embodiments of the present inventionwill be described.

FIG. 1 is a diagram illustrating a microscope system according to anembodiment of the present invention. FIG. 2 is a diagram illustrating aconfiguration of an optical system included in the microscope system.FIG. 3 is a diagram illustrating a configuration of a slide set. Themicroscope system 1 illustrated in FIG. 1 uses a slide set prepared inadvance to provide a user with appropriate information at a necessarytiming while the user works under a microscope 100 while looking into anocular lens 106. A configuration of the microscope system 1 will bedescribed with reference to FIGS. 1 to 3 .

The microscope system 1 includes the microscope 100, a control device200, a monitor 300, a plurality of input devices 400 (mouse 401,keyboard 402, foot switch 403, barcode reader 404), and a web camera500.

The microscope 100 is a stereoscopic microscope that allows the user tostereoscopically view a sample, and includes a microscope optical system110 illustrated in FIG. 2 . The microscope optical system 110 is anoptical system for a stereoscopic microscope. The user can observe anoptical image formed on the object side of the ocular lens 106 (ocularlens 106 a, ocular lens 106 b) by the microscope optical system 110 withthe left and right eyes via the ocular lens 106, and canstereoscopically observe the sample. Therefore, the microscope 100 issuitable for applications such as the assembly work of precisionequipment, for example.

The microscope 100 includes a zoom lens 102 (zoom lens 102 a, zoom lens102 b) operable with a zoom handle 130. By operating the zoom handle130, it is possible to change the observation magnification whilelooking into the ocular lens 106 to continuously observe the sample.

The microscope 100 includes a focusing handle 140. By operating thefocusing handle 140, it is possible to change the distance between thesample and an objective 101 to focus on the sample.

The microscope 100 includes an imaging device 112 that images the sampleto acquire a digital image of the sample. An ocular tube 120 to whichthe ocular lens 106 is attached is a trinocular tube, and the imagingdevice 112 is attached to the ocular tube 120. The imaging device 112 isprovided with a two-dimensional image sensor. The image sensor is notparticularly limited, and is, for example, a CCD image sensor, a CMOSimage sensor, or the like. The digital image acquired by the imagingdevice 112 is output to the control device 200. Furthermore, the digitalimage may be directly output to the monitor 300.

As illustrated in FIG. 2 , light propagating from one of left and rightoptical paths of the microscope optical system 110 and branched by abeam splitter 103 a such as a half mirror enters the imaging device 112via an imaging lens 111. In order to compensate for the optical pathlengths of the left and right optical paths generated by the beamsplitter 103 a and to suppress the difference between light amounts, anND prism 103 b is provided in the other optical path.

The microscope 100 includes a projector 113 that projects an auxiliaryimage on an image plane on which an imaging lens 105 (imaging lens 105a, imaging lens 105 b) forms an optical image. The projector 113 is adevice that projects and superimposes the auxiliary image on the imageplane in accordance with a command from the control device 200. Morespecifically, the projector 113 superimposes the auxiliary image on theimage plane based on auxiliary image data to be described later. Notethat the type of the projector 113 is not particularly limited. Theprojector 113 may be configured with, for example, a liquid crystaldevice or a digital mirror device.

The projector 113 is provided in the ocular tube 120. Light from theprojector 113 is guided to the left and right optical paths of themicroscope optical system 110 via a projection lens 114 and a pluralityof beam splitters (beam splitter 115, beam splitter 104 a, and beamsplitter 104 b).

As illustrated in FIG. 1 , the ocular tube 120 is provided with anoperation unit 121. The user can switch on and off the projector 113 byoperating the operation unit 121, and can instruct to start or stop thesuperimposition of the auxiliary image on the image plane.

The control device 200 controls the microscope 100. The control device200 generates the auxiliary image data described above and outputs theauxiliary image data to the microscope 100 (projector 113). Theauxiliary image data is generated using a slide set 10 stored in advancein the control device 200 as illustrated in FIG. 3 .

The slide set 10 includes a plurality of slides (slide 11, slide 12,slide 13, slide 14, slide 15, slide 16). The slide set 10 is, forexample, information for supporting assembly of precision equipment, andmore specifically, may be a procedure manual for the assembly work. Theplurality of slides are arranged in advance. That is, the slide set 10includes the plurality of slides ordered in advance.

Each slide includes one or more pieces of content to be projected ontothe image plane as an auxiliary image. Each of the plurality of slidesis information on each process of assembling the precision equipment,and more specifically, may be information including the content of work,cautions, and the like in each process.

The control device 200 generates an auxiliary image based on informationregarding a target slide selected from among the plurality of slidesincluded in slide set 10. The auxiliary image includes the target slide.The control device 200 appropriately switches the target slide so thatthe microscope system 1 can switch the auxiliary image projected on theimage plane so as to provide appropriate information to the user.

The monitor 300 and the input devices 400 are connected to the controldevice 200. The monitor 300 is, for example, a liquid crystal display,an organic EL display, or the like. The web camera 500 transmits acaptured image to the control device 200 via a network such as theInternet. The web camera 500 images, for example, a user who uses themicroscope system 1.

The microscope system 1 having such a configuration as described aboveperforms an image projection process illustrated in FIG. 4 . FIG. 4 isan example of a flowchart of the image projection process to beperformed by the microscope system. FIG. 5 is a diagram for describing aconfiguration of an image formed on the image plane. The imageprojection process performed by the microscope system 1 will bedescribed below with reference to FIGS. 4 and 5 .

First, the microscope system 1 projects an optical image of the sampleonto the image plane (step S1). Here, the imaging lens 105 focuses lightfrom the sample captured by the objective 101 onto the image plane toform an optical image of the sample. As a result, for example, anoptical image A1 in FIG. 5 is projected on the image plane.

Next, the microscope system 1 selects a target slide (step S2). Here,the control device 200 selects the target slide from the slide set 10.As long as it is immediately after the start of the image projectionprocess illustrated in FIG. 4 , the control device 200 may select, forexample, the first slide 11 from among the plurality of ordered slidesincluded in slide set 10 as the target slide. When detecting aninstruction to switch the target slide, the control device 200 mayselect the next slide (for example, the slide 12) as a new target slidein a case where the instruction is an instruction to switch to the nextslide, and may select the previous slide (for example, the slide 16) asthe new target slide in a case where the instruction is an instructionto switch to the previous slide. That is, the control device 200 selectsthe slide determined according to the order in which the plurality ofslides are ordered as the new target slide in response to theinstruction to switch the target slide.

For example, the user may input the instruction to switch the targetslide to the control device 200 using any one or more of the inputdevices. In addition, the instruction to switch the target slide may begenerated by the control device 200 itself, for example.

For example, the control device 200 may generate the switchinginstruction based on an elapsed time measured by a timer. Furthermore,the control device 200 may generate the switching instruction based onan analysis result of an image captured by the imaging device 112. Inthis case, the control device 200 may select the slide determinedaccording to the analysis result of the image captured by the imagingdevice 112 as the new target slide. Furthermore, the control device 200may generate the switching instruction based on, for example, a gestureof the user recognized via the web camera 500.

When the target slide is selected, the microscope system 1 generatesauxiliary image data (step S3). Here, the control device 200 generatesthe auxiliary image data based on information regarding the target slideselected in step S2. For example, when the slide 11 illustrated in FIG.3 is selected as the target slide, the control device 200 generatesauxiliary image data corresponding to an auxiliary image B1 illustratedin FIG. 5 and including the target slide (slide 11) based on informationsuch as the type (rectangular gauge) of the content included in thetarget slide, the size (width and height) of the content, the position(coordinates) of the content, and other options (presence or absence ofdimension display).

Finally, the microscope system 1 projects the auxiliary image onto theimage plane (step S4). Here, the control device 200 outputs theauxiliary image data generated in step S3 to the microscope 100, and theprojector 113 of the microscope 100 projects the auxiliary image B1 ontothe image plane based on the auxiliary image data output from thecontrol device 200. Consequently, as illustrated in FIG. 5 , theauxiliary image B1 is superimposed on the image plane on which theoptical image A1 is formed, and a superimposed image C1 in which theauxiliary image B1 is superimposed on the optical image A1 is formed.

As described above, in the microscope system 1, the slide selected fromthe slide set prepared in advance is projected as the auxiliary image onthe image plane on which the optical image is formed. Further, the slideprojected on the image plane is switched according to a predeterminedorder according to the switching instruction. For this reason, even inthe case of performing work such as the assembly work in which necessaryinformation differs for each process, the user can obtain theinformation corresponding to the work process while observing theoptical image of the sample without releasing the eyes from the ocularlens 106 by switching the slide. As a result, it is possible to performa series of work such as the assembly work without frequently moving theline of sight between the ocular lens 106 and the monitor 300.Therefore, according to the microscope system 1, the efficiency of theuser's work performed under the microscope can be greatly improved.

Functions of a software application (hereinafter referred to as a worksupport application) that is provided by the microscope system 1 andsupports assembly work performed under the microscope will be describedbelow more specifically. FIG. 6 is a diagram illustrating an example ofa home screen displayed on the monitor. In the microscope system 1, whenthe control device 200 executes a predetermined program, the worksupport application is started, and a window W1 illustrated in FIG. 6 isdisplayed on the monitor 300.

Immediately after the work support application is started, the homescreen illustrated in FIG. 6 is displayed in the window W1 in a statewhere a home tab is selected. From the home screen, the user can selecta work mode for supporting the assembly work, a training mode forreceiving instruction from a trainer, and a procedure manual mode forcreating a procedure manual to be used in the assembly work.

Furthermore, various settings can be adjusted by selecting a setup tab.Various settings for appropriately operating the work supportapplication will be described below with reference to FIGS. 7 to 10 .

FIG. 7 is a diagram illustrating an example of a screen for setting amicroscope configuration. FIG. 8 is a diagram illustrating an example ofa screen for adjusting AR display. FIG. 9 is a diagram for describing amethod of adjusting the AR display. FIG. 10 is a diagram illustrating anexample of a screen for adjusting a zoom sensor.

When a microscope configuration tab in the setup tab is selected, thescreen illustrated in FIG. 7 is displayed. The user can cause thecontrol device 200 to correctly recognize the configuration of themicroscope 100 by selecting each of the zoom lens body, the objective,an intermediate tube, a camera adapter, and the camera from a pull-downlist based on the configuration of the microscope 100 according to aprocedure displayed on the screen illustrated in FIG. 7 . As a result,the control device 200 can also recognize, for example, magnification(magnification excluding zoom magnification) serving as a reference ofthe microscope optical system 110.

When an AR display adjustment tab in the setup tab is selected, thescreen illustrated in FIG. 8 is displayed. By adjusting AR displayaccording to a procedure displayed on the screen illustrated in FIG. 8 ,the user can match a position on the digital image acquired by theimaging device 112 with the position of the AR display. As a result, theAR, that is, each piece of content of the slide included in theauxiliary image can be correctly displayed at the position recognized bythe control device 200 via the digital image.

Note that a specific procedure of the AR display adjustment is asfollows. First, the user (1) places a sample for adjustment on a stage,and (2) looks into the ocular lens 106. During the AR displayadjustment, an auxiliary image B2 including a digital image acquired bythe imaging device 112 is projected on the image plane by the projector113. Therefore, the user can check a superimposed image C2 asillustrated in FIG. 9 through the ocular lens 106. The superimposedimage C2 is an image on which the auxiliary image B2 is superimposed onan optical image A2 of the sample for adjustment. The auxiliary image B2includes a digital image (content B2 a) of the sample for adjustment andan adjustment menu (content B2 b).

Thereafter, the user (3) performs adjustment so that the AR displayoverlaps the sample for adjustment. Specifically, by operating theadjustment menu to adjust the projection position, angle, and size ofthe digital image, the digital image of the sample and the optical imageof the sample are exactly matched on the image plane. When theadjustment is successfully completed, the user (4) presses aregistration button. As a result, setting information for appropriatelyprojecting the auxiliary image onto the image plane is recorded in thecontrol device 200 based on the information of the display position (X,Y), the angle, and the size after the adjustment. Specifically, thissetting information is a conversion formula for converting the pixelposition of the imaging device 112 into the pixel position of theprojector 113. This conversion formula is calculated from theinformation of the display position, angle, and size after theadjustment described above. In FIGS. 8 and 9 , the example including theabove three elements has been described, but the conversion formula mayinclude at least one or more of elements, a size (scaling), rotation,translation, and a distortion coefficient.

When a zoom sensor adjustment tab in the setup tab is selected, thescreen illustrated in FIG. 10 is displayed. The user can cause thecontrol device 200 to correctly recognize the setting of the zoom lens102 by adjusting the zoom sensor according to a procedure displayed onthe screen illustrated in FIG. 10 . As a result, since the controldevice 200 can correctly recognize the zoom magnification changed byoperating the zoom handle 130 via the zoom sensor, the currentmagnification of the microscope optical system 110 can be correctlyrecognized from the reference magnification and the zoom magnificationof the microscope optical system 110.

Note that a specific procedure of the zoom sensor adjustment is asfollows. The user (1) adjusts a zoom dial to each click position (0.8×,2.0×, 3.2×, 4.0×, 5.6×) by operating the zoom handle 130, and thenpresses an acquisition button. As a result, the control device 200acquires sensor information generated at each magnification from thezoom sensor. Next, the user (2) instructs the execution of theadjustment. Here, the control device 200 updates information indicatingthe relationship between the sensor information output from the zoomsensor and the zoom magnification based on the sensor informationacquired in (1) and information of the zoom magnification correspondingto the sensor information. Finally, the user (3) checks the adjustmentresult. Here, the user turns the zoom dial to an arbitrary position tocheck whether the zoom magnification indicated by the dial matches amagnification sensor value displayed on the screen.

Next, the work mode of the work support application will be described.FIG. 11 is a diagram illustrating an example of the home screen observedfrom the ocular lens. FIG. 12 is an example of a flowchart of anassembly work support process.

The work mode is started when the user presses an assembly work startbutton on the home screen displayed on the monitor 300 as illustrated inFIG. 6 . In addition, when the user performs a predetermined operation(for example, pressing an allocation key) using any one or more of theinput devices 400, the projector 113 projects an auxiliary image B3illustrated in FIG. 11 on the image plane, and a menu screen isdisplayed in the field of view. The work mode may be started when theuser selects “Open” on the menu screen (auxiliary image B3) observedthrough the ocular lens 106. When the work mode is started, the controldevice 200 starts the assembly work support process illustrated in FIG.12 .

First, the control device 200 reads a slide file (step S11). Morespecifically, the control device 200 reads the slide file selected bythe user using any one or more of the input devices 400. The slide fileis a file in which information of the slide set is recorded, and iscreated and edited in the procedure manual mode described later.

The slide file is typically selected by the user clicking an icon of theslide file in the folder, but the selection method is not limited tothis method. For example, the barcode reader 404 may acquireidentification information (for example, the path of the slide file) ofthe slide set from a barcode, and the control device 200 may read theslide file based on the identification information acquired by thebarcode reader 404 to acquire the slide set. That is, the barcode reader404 is an example of an acquisition device that acquires theidentification information of the slide set, and the control device 200may acquire the slide set based on the identification informationacquired by the acquisition device.

The acquisition device that acquires the identification information ofthe slide set is not limited to a one-dimensional code reader such asthe barcode reader 404, and may be a two-dimensional code reader such asa QR code reader (QR code is a registered trademark) or an RF tagreader. Furthermore, for example, the acquisition device may detect atwo-dimensional code from the digital image acquired by the imagingdevice 112 to acquire the identification information. That is, theacquisition device may include at least one of a one-dimensional codereader, a two-dimensional code reader, an RF tag reader, and an imagingdevice.

When the slide file is read, control device 200 selects the first slideas the target slide (step S12). More specifically, the control device200 selects the first slide as the target slide from among the pluralityof ordered slides included in the slide set corresponding to the slidefile selected in step S1.

Next, the control device 200 generates auxiliary image data (step S13).More specifically, the control device 200 generates the auxiliary imagedata based on information regarding the target slide (the first slide)selected in step S12.

In a case where the size of the auxiliary image projected on the imageplane is changed according to the observation magnification, the controldevice 200 may generate the auxiliary image data based on theinformation regarding the target slide and the magnification informationof the microscope optical system 110. Since the above-describedadjustment of the zoom sensor is performed in advance, the controldevice 200 can easily acquire accurate magnification informationregarding the microscope optical system 110.

When the auxiliary image data is generated, the control device 200controls the projector 113 as the superimposing device such that anauxiliary image is superimposed on the image plane (step S14). In a casewhere the whole flow of the assembly work is itemized in text in thefirst slide, for example, as illustrated in FIG. 13 , a superimposedimage C4 in which an auxiliary image B4 of text content is superimposedon the optical image A1 is formed on the image plane. As a result, theuser can check at a glance the content of the work to be performed atthe initial stage of the assembly work by using the auxiliary image B4displayed in the field of view. Note that FIG. 13 is a diagramillustrating an example of a superimposed image observed from the ocularlens.

Thereafter, the control device 200 determines whether or not a switchinginstruction has been detected (step S15). For example, the user caninput the switching instruction to the control device 200 using any oneor more of the input devices 400. FIG. 14 is a diagram for describing aslide switching operation. For example, the switching instruction may beinput to the control device 200 by pressing a switch (switch 131, switch132) provided on the zoom handle 130 as illustrated in FIG. 14 . Aninstruction to switch to the next slide may be input to the controldevice 200 by pressing the switch 131, and an instruction to switch tothe previous slide may be input to the control device 200 by pressingthe switch 132.

The switching instruction may be input to the control device 200 by therotation of the wheel of the mouse 401. An instruction to switch to thenext slide may be input to the control device 200 by rotating the wheelforward, and an instruction to switch to the previous slide may be inputto the control device 200 by rotating the wheel backward. Alternatively,the switching instruction may be input to the control device 200 bypressing a shortcut key of the keyboard 402. Further, the switchinginstruction may be input to the control device 200 by pressing the footswitch 403. An instruction to switch to the next slide may be input tothe control device 200 by stepping on the right region of the footswitch 403, and an instruction to switch to the previous slide may beinput to the control device 200 by stepping on the left region of thefoot switch 403. Further, the switching of the slide may be instructedby voice, and a microphone (not illustrated) may function as an inputdevice that inputs the switching instruction. That is, the input devicethat inputs the switching instruction may include at least one of amouse, a keyboard, a switch provided on a handle, a foot switch, and amicrophone. It is desirable that the input device that inputs theswitching instruction be able to instruct the switching by one simpleoperation that the user can perform while looking into the ocular lens106.

When detecting the switching instruction (YES in step S15), the controldevice 200 selects a new target slide (step S16). Here, when detectingthe instruction to switch to the next slide, the control device 200selects the next slide of the current target slide as the target slideaccording to the order of the plurality of slides from a plurality ofslide sets included in the slide set acquired in step S11. Similarly,when detecting the instruction to switch to the previous slide, thecontrol device 200 selects the previous slide of the current targetslide as the target slide according to the order of the plurality ofslides from among the plurality of slide sets.

Thereafter, the control device 200 performs the processing of steps S13and S14 again. In a case where the new target slide includes a textdescribing the cautions in the work in the process to be performed, forexample, a superimposed image C5 in which an auxiliary image B5including text content is superimposed on the optical image A1 asillustrated in FIG. 15 is formed on the image plane. As a result, it ispossible to cause the user to check the cautions in the work immediatelybefore the work, and thus, it is possible to expect an effect ofreducing human errors caused in the work. Note that FIG. 15 is a diagramillustrating an example of a superimposed image observed from the ocularlens.

The control device 200 repeatedly performs the processing of steps S13to S16 in response to the input of the switching instruction. As aresult, for example, a plurality of slides included in a slide set aresequentially projected on the image plane, and information to beprovided is provided to the user at a necessary timing. In addition,even in a case where the user redoes the work, the slide set is returnedand appropriate information is provided to the user.

FIGS. 16 to 28 are diagrams illustrating other examples of thesuperimposed image observed from the ocular lens. With reference toFIGS. 16 to 28 , an example of the assembly work performed by the userwhile switching the slides will be described below.

FIGS. 13 and 15 illustrate a case where a slide including text contentis selected as the target slide. However, the slide may include graphiccontent such as a square, a circle, or a straight line. FIG. 16illustrates an example in which an auxiliary image B6 includingrectangular graphic content B6 a and circular graphic content B6 b isprojected on the image plane. The auxiliary image B6 may be used foralignment of the sample, for example. The user may move the stage suchthat the content included in the auxiliary image B6 overlaps apredetermined structure in the sample while observing a superimposedimage C6 in which the auxiliary image B6 is superimposed on the opticalimage A1.

When the alignment using the auxiliary image B6 is completed, the usermay instruct switching of the slide to inspect whether the size of thespecific structure in the sample meets the specification. For example,FIG. 17 illustrates an example in which a superimposed image C7 in whichan auxiliary image B7 including rectangular gauge content issuperimposed on the optical image A1 is formed on the image plane. Thegauge content may include vertical and horizontal dimension indicationsas illustrated in FIG. 17 . The gauge content is created in advance in asize corresponding to the specification. The user may compare the gaugecontent included in the auxiliary image B7 with the specific structureof the sample appearing in the optical image to inspect whether the sizeof the specific structure meets the specification.

The user may change the observation magnification by operating the zoomhandle 130 during the inspection. By observing at a highermagnification, it may be carefully inspected whether the specificstructure of the sample meets the specification. FIG. 18 illustrates asuperimposed image C8 in which an auxiliary image B8 is superimposed onan optical image A8 of the sample enlarged at a higher magnificationthan the optical image A1. The auxiliary image B8 includes gauge content(content B8 a) enlarged in accordance with the optical image A8.

Further, as illustrated in FIG. 18 , the auxiliary image B8 may includesetting information indicating the current magnification of themicroscope 100 as text content (content B8 b) in addition to the gaugecontent (content B8 a). Note that, although FIG. 18 illustrates anexample in which the auxiliary image B8 includes the total magnificationand the zoom magnification of the microscope 100, only one of thesemagnifications may be included in the auxiliary image B8. In addition,the magnification (for example, the magnification of the objective, thezoom magnification, the magnification of the intermediate tube, themagnification of the ocular lens, and the like) of each optical systemthat affects the total magnification may be projected on the imageplane. As a result, it is possible to recognize the set magnificationwhile performing observation during the work. In particular, since themagnification that the user adjusts while looking into the ocular lens,such as the zoom magnification, is displayed as the auxiliary image, itis not necessary to take the eyes off the ocular lens only to check themagnification, which greatly contributes to the improvement of the workefficiency. Note that the setting information may be displayed at apredetermined timing, for example, after an operation of changing themagnification. Furthermore, the setting information is not limited tothe information regarding the magnification, and may be informationregarding a current arbitrary setting of the microscope 100. Forexample, the type of the objective, the setting of a filter, and thelike may be included in the auxiliary image as the setting information.

The projection of the auxiliary image including the content having thesize according to the zoom magnification can be performed by the controldevice 200 generating the auxiliary image data based on themagnification information of the microscope optical system 110 and theinformation regarding the target slide. More specifically, the controldevice 200 may generate the auxiliary image data such that predeterminedcontent among one or more pieces of content included in the target slideis included in the auxiliary image in a size corresponding to themagnification information. The predetermined content is the content ofthe first classification, and is, for example, a reticle or the like inaddition to the above-described gauge. Since the gauge and the reticleare used for dimension measurement, it is desirable that the gauge andthe reticle be projected by changing the sizes according to theobservation magnification.

On the other hand, the control device 200 may generate the auxiliaryimage data such that content other than the predetermined content amongthe one or more pieces of content included in the target slide isincluded in the auxiliary image in a predetermined size. The contentother than the predetermined content is the content of the secondclassification, and is, for example, text content. It is desirable thatthe text be displayed at a constant ratio with respect to the fieldnumber, that is, at a constant size regardless of the observationmagnification.

It is desirable that the control device 200 classify the contentincluded in the slide into, for example, the content of the firstclassification or the content of the second classification based on theinformation regarding the type of the content, and generate theauxiliary image data such that the size varies according to theclassification result. Note that examples of the type of content includea pen, a figure, a text, a still image, a moving image, a gauge, areticle, an image analysis result, a timer, a dashboard, external deviceinformation, content for measurement, and the like.

FIG. 19 illustrates an example in which a superimposed image C9 in whichan auxiliary image B9 including cross-shaped reticle content issuperimposed on the optical image A1 is formed on the image plane.Similarly to the gauge content, the reticle content is the content ofthe first classification, and is enlarged or reduced according to thezoom magnification and projected. The user may switch the slide tochange the state to a state in which the auxiliary image B9 isprojected, and inspect the sample by recognizing the size of thespecific structure with the scale of the reticle included in theauxiliary image B9.

FIG. 20 illustrates an example in which a superimposed image C10 inwhich an auxiliary image B10 including text content (content B10 a),content (content B10 b) for line measurement, and text content (contentB10 c) is superimposed on the optical image A1 is formed on the imageplane. The user may switch the slide to change the state to a state inwhich the auxiliary image B10 is projected, measure the size of thepredetermined structure, and determine the pass or fail of theinspection. As illustrated in FIG. 20 , the user may use an input devicesuch as the mouse 401 to designate a range to be measured so as toinstruct measurement, and the control device 200 may cause the projector113 to project the auxiliary image B10 including the measurement resulton the image plane. Such dynamic display can be performed by the controldevice 200 generating the auxiliary image data based on positioninformation input by the user in addition to the magnificationinformation of the microscope optical system 110 and the informationregarding the target slide. Note that not only the content formeasurement but also content for surface measurement may be included inthe auxiliary image. The setting information (magnification) displayedas the text content is particularly effective when the magnification isdesignated as a requirement at the time of measurement. Since themagnification is included in the auxiliary image, the user can checkthat the magnification requirement designated in the text content issatisfied without taking his/her eyes off the ocular lens, and can inputthe measurement instruction.

In addition, the slide may include a still image or a moving image ascontent. FIG. 21 illustrates an example in which a superimposed imageC11 in which an auxiliary image B11 including a still image issuperimposed on an optical image A11 is formed on the image plane. Theuser may perform the work while checking the state of the sample afterthe work is correctly performed on the still image, that is, the sample.FIG. 22 illustrates an example in which a superimposed image C12 inwhich an auxiliary image B12 including a moving image is superimposed onthe optical image A11 is formed on the image plane. The user may performthe work while checking the correct work procedure in the moving image.

The image displayed on the image plane as the auxiliary image may be animage captured in real time by the web camera 500. As a result, the usercan perform the work while checking both the micro image (optical imageof the sample) and the macro image (auxiliary image) obtained by imagingthe periphery of the microscope 100, the user's hand, or the like withthe web camera 500. FIG. 23 illustrates an example in which asuperimposed image C121 in which an auxiliary image B121 including animage obtained by imaging tools placed around the microscope 100 withthe web camera 500 is superimposed on the optical image A11 is formed onthe image plane. Furthermore, FIG. 24 illustrates an example in which asuperimposed image C122 in which an auxiliary image B122 including animage obtained by simultaneously imaging a work area and tools placedaround the work area with the web camera 500 is superimposed on theoptical image A11 is formed on the image plane. As illustrated in FIGS.23 and 24 , the peripheral tools are shown as auxiliary images, so thatthe tools can be smoothly switched.

In addition, the slides may include information of a timer or othersensors as content. FIG. 25 illustrates an example in which asuperimposed image C13 in which an auxiliary image B13 including a stillimage (content B13 a) and an elapsed time from the start of the work(content B13 b) is superimposed on the optical image A11 is formed onthe image plane. By checking the elapsed time with the content B13 b,the user may perform the work while checking whether it takes too muchtime. FIG. 26 illustrates an example in which a superimposed image C14in which an auxiliary image B14 including a still image (content B14 a)and a temperature (content B14 b) of the work area measured by athermometer (not illustrated) is superimposed on the optical image A1 lis formed on the image plane. The user may perform the work whilechecking that the temperature of the work area is within an appropriaterange in the content B14 b.

The last slide may include text content indicating a work result. FIG.27 illustrates an example in which a superimposed image C15 in which anauxiliary image B15 including text content is superimposed on theoptical image A11 is formed on the image plane. The user may check theprogress and results of the assembly work with the auxiliary image B15.The control device 200 may generate auxiliary image data correspondingto the auxiliary image B15 by communicating with another device tocollect the information.

Note that the control device 200 controls the projector 113 to displaythe plurality of slides included in the slide set selected by the userin the order determined according to the switching instruction, but maycontrol the projector 113 to display a specific slide by interruption.For example, when a manufacturing line which the user is in charge of ispaused, the microscope system 1 may notify such information by theauxiliary image. FIG. 28 illustrates an example in which a superimposedimage C16 in which an auxiliary image B16 including dashboard contentindicating the state of the manufacturing line is superimposed on theoptical image A11 is formed on the image plane. The user may check thestate of the manufacturing line with the auxiliary image B16.

FIG. 29 is a diagram illustrating an example of an imaging screenobserved from the ocular lens. FIG. 30 is a diagram illustrating anexample of an imaging screen observed on the monitor. FIGS. 31 to 33 arediagrams illustrating recording images. FIG. 34 is a diagram fordescribing an image recording method. A method of creating a work recordincluding an image will be described below with reference to FIGS. 29 to34 .

At the time of the assembly work, the user may instruct to image orrecord the sample, or may record a still image or a moving image asevidence of the work in the control device 200. The user may input animage capturing or recording instruction while looking into the ocularlens 106. For example, when a camera button is pressed on the homescreen illustrated in FIG. 11 , an auxiliary image B17 illustrated inFIG. 29 is projected on the image plane, and the imaging screen isdisplayed. The user may input an imaging instruction while viewing alive image in the auxiliary image B17 illustrated in FIG. 29 . The liveimage may be acquired by the imaging device 112, for example. Byinputting the imaging instruction while checking the live image, focusadjustment, brightness adjustment, and the like can be performed beforeimaging.

Note that the imaging instruction may be input from a window W2displayed on the monitor 300 as illustrated in FIG. 30 . FIG. 30illustrates a state in which a live image LV acquired by the imagingdevice 112 and the auxiliary image (content B10 a and content B10 b)projected on the image plane are displayed in a field-of-view mark FVindicating the field of view of the microscope 100. The user may inputan imaging instruction from the window W2.

When detecting an imaging instruction or a recording instruction (YES instep S17), the control device 200 controls the imaging device 112 toperform imaging or recording (step S18). Here, the control device 200causes a recording device to record a digital image (captured image) ofthe sample captured by the imaging device 112 in association with theauxiliary image projected on the image plane at the time of the imagecapturing. Note that the recording device may be provided in the controldevice or may be provided on a server placed on a network.

For example, as illustrated in FIG. 31 , the control device 200 maycreate a recording image F1 in which a captured image D1 and anauxiliary image E1 are synthesized, and record the recording image F1 inthe recording device. Further, the control device 200 may record thecaptured image D1 and the auxiliary image E1 as separate files, andadditionally record information associating the captured image D1 withthe auxiliary image E1. When only a part of the auxiliary image E1 isincluded in the field of view of the imaging device 112, the auxiliaryimage E1 may be recorded after only the part included in the field ofview is trimmed.

In FIG. 31 , the configuration of the recording image has been describedby exemplifying a case where the field of view of the imaging device 112is narrower than the field of view of the microscope optical system 110.In a case where the field of view of the imaging device 112 is widerthan the field of view of the microscope optical system 110, asillustrated in FIG. 32 , the control device 200 may create a recordingimage F2 in which a field-of-view mark FV indicating the field of viewof the microscope optical system 110 is synthesized in addition to acaptured image D2 and an auxiliary image E2, and record the recordingimage F2 in the recording device. The control device 200 may record thecaptured image D2, the auxiliary image E2, and the field-of-view mark FVas separate files, and additionally record information associating thecaptured image D2, the auxiliary image E2, and the field-of-view mark FVwith each other.

Furthermore, as illustrated in FIG. 33 , the control device 200 maycreate image data of a recording image F3 obtained by synthesizing anannotation image H3 in addition to the captured image D2, the auxiliaryimage E2, and the field-of-view mark FV, and record the image data inthe recording device. The annotation image H3 is projected usingannotation image data created in response to a drawing instruction inputby the user using any one or more of the input devices 400. In thisrespect, the annotation image H3 is different from the auxiliary imageE2 based on the slide. However, the annotation image H3 is similar tothe auxiliary image E2 in that the annotation image H3 is superimposedon the image plane by the projector 113 based on the image datagenerated by the control device 200.

The user may input the drawing instruction while looking into the ocularlens 106. For example, characters, figures, and the like may be drawn onthe image plane by pressing a pen button on the home screen illustratedin FIG. 11 . Furthermore, characters, figures, and the like may be addedon the window W2 illustrated in FIG. 30 . In this case, when charactersare written as content on the live image LV with a pen, the content isprojected at a position on the optical image that matches a position onthe live image LV where the content is placed. The content may be agraphic such as a line, an arrow, a square, or a circle, or may be atext, an image (still image, moving image), or the like, in addition toa character written with a pen. The user can leave notes and comments onthe work result using such a drawing function. The control device 200may record the captured image D2, the auxiliary image E2, and theannotation image H3 as separate files, and additionally recordinformation associating the captured image D2, the auxiliary image E2,and the annotation image H3 with each other.

Further, as illustrated in FIG. 34 , the control device 200 may recordan image G4 captured by the web camera 500 in association with arecorded image F4 in the recording device. By recording the image G4 inassociation with the recorded image F4, the state of the user at thetime of the work can also be recorded. The image G4 and the recordedimage F4 may be recorded as one image.

Note that the example has been described in which the recorded image isacquired at the end of the assembly work and used as evidence of thework performed by the user. However, the work record may be acquired inthe middle of the assembly work or may be acquired a plurality of timesat any timing during the work. In addition, although the still image isexemplified as the recorded image, the recorded image may be a movingimage, and the control device 200 may record the work from the start tothe end of the work in the moving image. For example, the image G4 andthe recorded image F4 illustrated in FIG. 34 may be recorded as a movingimage and may be synthesized and recorded as one image. As a result, theimage G4 and the recorded image F4 can be simultaneously reproduced onthe same time axis, and the relationship between the content of the workunder the microscope and the state of the user can be easily recognized.Therefore, the analysis of the work is also facilitated. In addition, inthe case of recording a moving image, chapter information linked to apage of a slide reproduced as an auxiliary image may be written in themoving image. Therefore, the searchability is improved when the movingimage is checked later.

In addition, the work record is not limited to an image, and may beother information. As the work record, the control device 200 may recordinformation not to be projected on the image plane, for example, settinginformation of the microscope system 1 together with an image as metainformation. In addition, the control device 200 may record voice data,an operation log of the microscope system 1, or the like as a workrecord. In addition, the control device 200 may record, as a workrecord, information such as time information acquired from a time serveror a user name used for logging in to the work support application, ormay record, as a work record, information (a component name or a serialnumber) for identifying the sample. The work record is not limited to acase where the user explicitly instructs recording, and the controldevice 200 may automatically record the work record.

When an end instruction is input after the imaging instruction (YES instep S19), the control device 200 ends the assembly work support processillustrated in FIG. 12 .

When the microscope system 1 performs the assembly work support processillustrated in FIG. 12 , the user can appropriately switch the slideaccording to the work process. Therefore, necessary information can beobtained at a necessary timing without taking the eyes off the ocularlens 106. Therefore, according to the microscope system 1, the work ofthe user can be greatly improved in efficiency.

Next, a training mode of the work support application will be described.The training mode is started when the user presses a training startbutton on the home screen displayed on the monitor 300 as illustrated inFIG. 6 . The training mode is different from the work mode in that aslide file created for training is used in the training mode, but is thesame as the work mode in other points. Therefore, also in the trainingmode, the control device 200 performs the assembly work support processillustrated in FIG. 12 .

In the training mode, a trainee who looks into the ocular lens 106 toperform work and a trainer who gives an instruction to the trainee whileviewing an image displayed on the monitor 300 use the microscope system1. The trainer can give an instruction to the trainee in real time bycausing the projector 113 to project the annotation image using thedrawing function used for evidence creation in the work mode.

Note that the trainer may give an instruction to the trainee from aremote location instead of operating the control device 200 using anyone or more of the input devices 400 to give an instruction to thetrainee. FIG. 35 is a diagram illustrating a configuration instructed bythe trainer from the remote location. As illustrated in FIG. 35 , thetrainer may access the microscope system 1 from a remote terminal(remote terminal 601, remote terminal 602) connected to the microscopesystem 1 via a network such as the Internet. For example, the trainermay access the control device 200 from the remote terminal using aremote desktop function or the like, and directly operate the controldevice 200 to give an instruction to the trainee. In addition, the worksupport application may be installed in the remote terminal to give aninstruction from the remote terminal to the trainee.

FIG. 36 is an example of a flowchart of a process of creating a slideset. FIG. 37 is a diagram illustrating an example of a screen forsetting a parameter of a slide file. FIG. 38 is a diagram illustratingan example of a slide setting screen. The procedure manual mode of thework support application will be described below with reference to FIGS.36 to 38 .

The procedure manual mode is started when the user presses a newcreation button for creating a procedure manual or an edit button on thehome screen displayed on the monitor 300 as illustrated in FIG. 6 . Whenthe new creation button is pressed and the procedure manual mode isstarted, the control device 200 starts the process of creating a slideset as illustrated in FIG. 36 .

First, the control device 200 newly creates a slide file (step S21).When the slide file is created, the control device 200 sets a parameterof the slide file according to input from the user (step S22). Forexample, the user can designate the number of slides by adding ordeleting a slide in a window W3 illustrated in FIG. 37 . In addition,the order of these slides can be designated by interchanging the slides.

After setting the parameter of the slide file, the control device 200creates slides one by one according to the input from the user.Specifically, the control device 200 repeats the setting of a background(step S23), the arrangement of content, and the setting of contentparameters (step S24) for each slide.

In step S23, the control device 200 sets a background to a slideselected by the user from a region 20 of the window W3 illustrated inFIG. 38 . Specifically, when the user presses a button 31 in a region30, the control device 200 does not set anything for the background ofthe slide. When the user presses a button 32, the control device 200sets a live image acquired by the imaging device 112 on the backgroundof the slide. When the user presses a button 33, the control device 200sets a moving image acquired in advance on the background of the slide.When the user presses a button 34, the control device 200 sets a stillimage acquired in advance on the background of the slide. FIG. 38illustrates a state in which the still image acquired in advance isdisplayed in a region 40 displaying the slide selected in the region 20as a result of pressing the button 34 by the user.

Setting the live image on the background is particularly suitable in acase where the sample to be used in the working mode can be preparedwhen a slide set is to be created in the procedure manual mode. Bycreating a slide while displaying the sample to be actually used in theworking mode as the live image, it is possible to arrange content at anappropriate position and size according to the sample in step S24.

In addition, setting the still image or the live image on the backgroundis particularly suitable in a case where the sample to be used in theworking mode cannot be prepared in a case where the slide set is createdin the procedure manual mode. Even in a case where the sample cannot beprepared, by creating a slide with a still image or a moving imageobtained by imaging the sample in advance as the background, it ispossible to arrange content at an appropriate position and sizeaccording to the sample in step S24.

Note that the background set in step S23 is used for the user to easilyperform the work of arranging and setting the content in step S24. Thatis, the background set here is used only for creating and editingslides, and the background set here is not displayed in the auxiliaryimage projected in the work mode.

In step S24, the control device 200 arranges the content on the slide inwhich the background is set in step S23, and sets the parameters of thecontent. Specifically, the control device 200 arranges the content onthe slide according to the input from the user, and incorporates thecontent as a component of the slide.

The type of content to be arranged on the slide can be selected in aregion 50. When the type of content is selected from the region 50,further subdivided content (also referred to as an object) belonging tothe content can be selected from a region 60.

For example, when graphic content is selected in the region 50, contentsuch as a line, an arrow, a square, a circle, a text, an image (stillimage), and a video (moving image) can be selected from the region 60.In addition, when gauge content is selected in the region 50, forexample, content such as a linear gauge, a rectangular gauge, or acircular gauge can be selected from the region 60. Also, when reticlecontent is selected in the region 50, content such as, for example, across reticle, a grid reticle, or the like may be selected from theregion 60. FIG. 38 illustrates a state in which, as a result of the userselecting the gauge content in the region 50 and further selecting therectangular gauge in the region 60, the rectangular gauge is arranged asthe content in the region 40 displaying the slide selected in the region20.

The parameters of the content can be set in a region 70. As theparameters of the content, a height, a width, a rotation (orientation),a vertical position, a horizontal position, and the like can be set. Inthe case of a gauge or a reticle, whether or not to display the lengthcan also be set as a parameter. Note that the parameters of the contentmay be set by moving or scaling the content in the region 40. Note thatparameters of each piece of content are not limited to those exemplifiedhere. Each piece of content may include other parameters such as color,line width, line type, and the like.

When the processing of steps S23 and S24 is completed for all theslides, the control device 200 stores the slide file (step S25), andends the process of creating a slide set illustrated in FIG. 36 . Asdescribed above, in the microscope system 1, the control device 200creates a new slide set according to the operation on the creationscreen displayed on the monitor 300 serving as the display device.

Note that although FIG. 36 illustrates the process of newly creating aslide set, the process of editing the slide set is basically similar tothe process illustrated in FIG. 36 . In a case where the slide file isedited, an existing slide file is read in the region 20 in step S21instead of newly creating a slide file, and the subsequent processing isperformed. That is, in the microscope system 1, the control device 200edits the existing slide set according to an operation on an editingscreen displayed on the monitor 300 serving as the display device.

As described above, according to the microscope system 1, the user canfreely create and edit the slide set to be used as the procedure manualby using the procedure manual mode. By using the slide set that has beencreated in the procedure manual mode and in which the user arranges thenecessary information in the necessary order, the microscope system 1can provide the user with appropriate information at a necessary timingduring the work under the microscope in the work mode.

FIG. 39 illustrates an exemplary hardware configuration of a computer200 a for implementing the control device 200 according to theembodiment described above. The computer 200 a having the hardwareconfiguration illustrated in FIG. 39 includes, for example, a processor201, a memory 202, a storage device 203, a reading device 204, acommunication interface 206, and an input/output interface 207. Notethat the processor 201, the memory 202, the storage device 203, thereading device 204, the communication interface 206, and the I/Ointerface 207 are connected to one another, for example, via a bus 208.

The processor 201 may be, for example, a single processor, amultiprocessor, or a multi-core processor. The processor 201 reads aprogram stored in the storage device 203 and executes the program,thereby performing the control processes illustrated in FIGS. 12, 36 ,and the like.

The memory 202 is, for example, a semiconductor memory and may include aRAM area and a ROM area. For example, the storage device 203 is a harddisk, a semiconductor memory such as a flash memory, or an externalstorage device.

For example, the reading device 204 accesses a removable recordingmedium 205 in accordance with an instruction of the processor 201. Forexample, the removable recording medium 205 is implemented by asemiconductor device, a medium to and from which information is inputand output by a magnetic effect, a medium to and from which informationis input and output by an optical effect, or the like. Note that thesemiconductor device is, for example, a Universal Serial Bus (USB)memory. Such a medium to and from which information is input and outputby a magnetic effect is, for example, a magnetic disk. Such a medium toand from which information is input and output by an optical effect is,for example, a compact disc (CD)-ROM, a digital versatile disc (DVD), ora Blu-ray disc (Blu-ray is a registered trademark).

The communication interface 206 communicates with other devices (forexample, the microscope 100, the web camera 500, and the like) accordingto an instruction of the processor 201, for example. The input/outputinterface 207 is, for example, an interface between the input devices400 and an output device. The input devices 400 are, for example,devices such as the mouse 401, the keyboard 402, the foot switch 403,and the like that receive an instruction from the user. The outputdevice is, for example, the monitor 300 and an audio device such as aspeaker.

For example, the program that the processor 201 executes is provided tothe computer in the following forms:

-   -   (1) Installed in the storage device 203 in advance    -   (2) Provided by the removable recording medium 205    -   (3) Provision from a server, such as a program server.

Note that the hardware configuration of the computer for implementingthe control device 200, described with reference to FIG. 39 , isexemplary and thus the embodiment is not limited to this. For example, apart of the configuration described above may be omitted or a newconfiguration may be added to the configuration described above.Furthermore, in another embodiment, for example, some or all of thefunctions of the above-described processing device may be implemented ashardware such as a field programmable gate array (FPGA), asystem-on-a-chip (SoC), an application specific integrated circuit(ASIC), a programmable logic device (PLD), or the like.

The embodiments described above are specific examples for facilitatingunderstanding of the invention, and thus the present invention is notlimited to the embodiments. Modifications of the embodiments describedabove and alternatives to the embodiments described above are to beincluded. That is, the constituent elements in each embodiment can bemodified without departing from the spirit and scope of the embodiment.A new embodiment can be implemented by appropriately combining aplurality of constituent elements disclosed in one or more of theembodiments. Some constituent elements may be omitted from theconstituent elements in each embodiment, or some constituent elementsmay be added to the constituent elements in each embodiment.Furthermore, the process procedure in each embodiment may be changed inorder as long as there is no contradiction. That is, the microscopesystem, the superimposing unit, and the operation method of the presentinvention can be variously modified and changed without departing fromthe scope of the claims.

In the above-described embodiment, the example in which the controldevice 200 that controls the operation of the entire microscope system 1generates the auxiliary image data has been described, but the auxiliaryimage data may be generated by a control device provided in the oculartube 120. That is, the ocular tube 120 attached to the microscope 100may function as a superimposing unit including a control device thatgenerates auxiliary image data based on information regarding a targetslide selected from among a plurality of ordered slides included in aslide set, and a superimposing device that superimposes, based on theauxiliary image data, an auxiliary image including the target slide onan image plane on which an optical image is formed.

Furthermore, although not particularly mentioned in the above-describedembodiment, the image plane (AR screen) of the microscope optical system110 on which an auxiliary image is projected may be a screen on whichthe content of a slide displayed on the monitor 300 is mirrored, or apart of video displayed on the monitor 300 may be projected on the imageplane as the auxiliary image. Furthermore, the mouse cursor displayed onthe monitor 300 may be projected onto the auxiliary image as a pseudomouse cursor. The pseudo mouse cursor does not need to be constantlyprojected, and for example, when the pseudo mouse cursor does not movefor a certain period of time, the pseudo mouse cursor may be deletedfrom the auxiliary image. Accordingly, it is possible to preventunnecessary display of the pseudo mouse cursor from hinderingconcentration of the user. Further, the position of the pseudo mousecursor does not necessarily match the position of the mouse cursordisplayed on the monitor 300. For example, when the mouse cursor on themonitor 300 moves out of the field of view of the microscope opticalsystem 110, the pseudo mouse cursor may be displayed at an end of thefield of view. As a result, it is possible to avoid a situation in whichthe user loses sight of the pseudo mouse cursor due to the movement ofthe pseudo mouse cursor to the outside of the field of view.

Although not particularly mentioned in the above-described embodiment,the auxiliary image may be generated using a learned model created bymachine learning such as deep learning. The slide set may include, forexample, a slide including content (hereinafter, AI content) obtained byinputting a digital image acquired by the imaging device 112 to thelearned model. Specifically, for example, a defect of the sample may bedetected from the digital image by using a learned model that haslearned the defect of the sample to be inspected, and as illustrated inFIG. 40 , an auxiliary image B18 including a bounding box indicating theposition of the defect may be output. In addition, the AI content is notlimited to content to be used for inspection purposes and may beintended for work support. For example, a region to be soldered may bedetected from the digital image by using a learned model that haslearned a region requiring soldering, and as illustrated in FIG. 41 , anauxiliary image B19 including a bounding box indicating the detectedregion may be output. Note that the shape of the bounding box is notparticularly limited, and may be a rectangle as illustrated in FIG. 40or a circle as illustrated in FIG. 41 . The slide set may include aplurality of slides including AI content corresponding to the workprocess, and the user may appropriately switch between the plurality ofslides to provide necessary information to the user using variouslearned models.

In the present specification, the expression “based on A” does notindicate “based on only A” but indicates “based on at least A” andfurther indicates “based partially on at least A”. That is, “based on A”may be “based on B in addition to A” or “based on part of A”.

What is claimed is:
 1. A microscope system comprising: a microscopeoptical system that includes an ocular lens and forms an optical imageof a sample on an object side of the ocular lens; a processor thatgenerates auxiliary image data based on information regarding a targetslide selected from among a plurality of ordered slides included in aslide set; and a superimposing device that superimposes, based on theauxiliary image data, an auxiliary image including the target slide onan image plane on which the optical image is formed, wherein theprocessor selects, in response to an instruction to switch the targetslide, a slide determined according to a first order in which theplurality of slides are ordered as a new target slide.
 2. The microscopesystem according to claim 1, wherein the processor generates theauxiliary image data based on magnification information of themicroscope optical system and information regarding the target slide. 3.The microscope system according to claim 2, wherein the target slideincludes one or more pieces of content, and the processor is furtherconfigured to: generate the auxiliary image data such that content of afirst classification among the one or more pieces of content is includedin the auxiliary image in a size according to the magnificationinformation, and generate the auxiliary image data such that content ofa second classification among the one or more pieces of content isincluded in the auxiliary image in a predetermined size.
 4. Themicroscope system according to claim 3, wherein the content of the firstclassification includes at least one of a gauge and a reticle.
 5. Themicroscope system according to claim 3, wherein the one or more piecesof content include at least one of a pen, a figure, a text, a stillimage, a moving image, a gauge, a reticle, an image analysis result, atimer, a dashboard, external device information, and content formeasurement.
 6. The microscope system according to claim 1, furthercomprising an input device that inputs the instruction to switch thetarget slide.
 7. The microscope system according to claim 6, wherein theinput device includes at least one of a mouse, a keyboard, a switchprovided on a handle, a foot switch, and a microphone.
 8. The microscopesystem according to claim 1, further comprising: an acquisition devicethat acquires identification information for identifying the slide set,wherein the processor acquires the slide set based on the identificationinformation acquired by the acquisition device.
 9. The microscope systemaccording to claim 8, wherein the acquisition device includes at leastone of a one-dimensional code reader, a two-dimensional code reader, anRF tag reader, and an imaging device.
 10. The microscope systemaccording to claim 1, further comprising: an imaging device thatcaptures an image of the sample; and a recording device, wherein theprocessor causes the recording device to record the image of the samplecaptured by the imaging device in association with the auxiliary image.11. The microscope system according to claim 1, wherein the processorgenerates annotation image data according to a drawing instruction, andthe superimposing device superimposes an annotation image correspondingto the annotation image data on the image plane.
 12. The microscopesystem according to claim 11, further comprising: an imaging device thatcaptures an image of the sample; and a recording device, wherein theprocessor causes the recording device to record the image of the samplecaptured by the imaging device in association with the auxiliary imageand the annotation image.
 13. The microscope system according to claim1, further comprising: an imaging device that captures an image of thesample, wherein the processor selects, as a new target slide, a slidedetermined according to an analysis result of the image of the samplecaptured by the imaging device.
 14. The microscope system according toclaim 1, further comprising: an ocular tube to which the ocular lens isattached, wherein the ocular tube includes an operation unit configuredto instruct start or stop of the superimposition of the auxiliary imageon the image plane.
 15. The microscope system according to claim 1,further comprising: a display device, wherein the processor edits theslide set according to an operation on an edit screen displayed on thedisplay device.
 16. The microscope system according to claim 1, furthercomprising: a display device, wherein the processor creates a new slideset according to an operation on a creation screen displayed on thedisplay device.
 17. The microscope system according to claim 1, whereinthe microscope optical system is an optical system for astereomicroscope.
 18. The microscope system according to claim 1,wherein the slide set is information for supporting assembly ofprecision equipment, and each of the plurality of slides is informationon each step of the assembly of the precision equipment.
 19. Themicroscope system according to claim 1, wherein the plurality of slidesinclude a slide including AI content obtained by inputting an image ofthe sample into a model learned by a machine learning.
 20. Asuperimposing unit attached to a microscope including a microscopeoptical system that forms an optical image of a sample on an object sideof an ocular lens, the superimposing unit comprising: a processor thatgenerates auxiliary image data based on information regarding a targetslide selected from among a plurality of ordered slides included in aslide set; and a superimposing device that superimposes, based on theauxiliary image data, an auxiliary image including the target slide onan image plane on which the optical image is formed, wherein theprocessor selects, in response to an instruction to switch the targetslide, a slide determined according to a first order in which theplurality of slides are ordered as a new target slide.
 21. An operationmethod of a control device that controls a microscope including asuperimposing device and a microscope optical system that forms anoptical image of a sample on an object side of an ocular lens, theoperation method comprising: causing a processor of the control deviceto select, in response to an instruction to switch a target slideselected from among a plurality of ordered slides included in a slideset, a slide determined according to a first order in which theplurality of slides are ordered as a new target slide; causing theprocessor to generate auxiliary image data based on informationregarding the new target slide; and causing the superimposing device tosuperimpose, based on the auxiliary image data, an auxiliary imageincluding the new target slide on an image plane on which the opticalimage is formed.